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Real-Time Retail Display System 

Related Application 

[01] A claim of priority is made to U.S. Provisional Patent Application Serial 
No. 60/406,167, filed by Dietz et al. on August 27, 2002, titled Projector Base 
Display System. 

Field of the Invention 

[02] This invention relates generally to retail display systems, and more 
particularly to retail display systems that dynamically adapt environments to 
consumer behavior. 

Background of the Invention 

[03] There has been considerable prior work in constructing displays, sensing 
consumer characteristics and behavior and data mining techniques for preference 
prediction. 

[04] Bodin, in U.S. Published Application 20030040922 describes a system 
that alerts a consumer to nearby products that match the previously stored 
preferences of the identified consumer. That system needs to know the explicit 
identity of the consumer in order to access the database, and inherently assumes 
that the preferences have not changed. 
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[05] Eldering, in U.S. Published Application 20030004810, describes a 
system that requires explicit knowledge of the consumer' s identity to determine the 
applicability of an advertisement. 

[06] Sone, in U.S. Published Application 20020035560, describes a system 
that uses an RFID tag to explicitly identify a consumer and then present 
advertisements in a preferential format. 

[07] Black, in U.S. Published Application, 20020138433, describes a system 
that detects signals from personal wireless devices in order to recognize the 
presence of explicitly identified individuals, and then puts up appropriate 
advertising on an ATM screen. 

[08] Bermel, in U.S. Published Application, 20010038034, describes a 
method that updates a display based on the presence of electronic cards that 
explicitly identify individuals in the vicinity. 

[09] Loof, in U.S. Patent 6,507,279, describes an integrated self-checkout 
system that can present information to identified individuals. That system requires 
the individual to be a member of an explicitly authorized group as listed in a 
database. 

[010] All of those prior art systems inherently depend upon stored data about 
explicitly identified individuals. 
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Summary of the Invention 

[01 1] The present invention provides a system that uses computer-driven 
displays to present consumers with a variety of content to positively influence their 
purchase decisions. Unlike conventional signage and in-store video systems, the 
displays are interactive and constantly changing based upon a current behavior of 
the consumers and historical trends. Because the system responds to normal 
shopping behavior, there is nothing for the consumer to learn to operate, nor is 
there any need to carry any special device or special identification such as credit 
cards. 

[012] A network of sensors provides real-time information to the system. These 
sensors determine both consumer characteristics and behavior. Other information 
sources can also be utilized to supplement this data. These can include sensors that 
acquire environmental data such as weather, date, time, pricing, cash register 
receipts, and traffic data. 

[013] Based upon the available data, the system generates a dynamic model of 
consumer response as a function of the content presented. The model is used to 
optimize the presented content in order to achieve predetermined goals, such as 
maximizing total profits. 
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Brief Description of the Drawings 

[014] Figure 1 is a block diagram of a real-time retail display system according 
to the invention; 

[015] Figure 2 is a flow diagram of a method for operating the system of Figure 
1; and 

[016] Figure 3 is a diagram of a retail environment where the system of Figure 
1 is used. 

Detailed Description of the Preferred Embodiment 
System Structure 

[017] Figure 1 shows a real-time retail display system 100 according to the 
invention. The system includes a processor 1 10 coupled to displays 101 and 
sensors 102. The processor is conventional in structure, and includes memory and 
input/output interfaces. The processor can also include a database and a network 
interface. 

[018] Each display can include one or more projectors to show still images or 
videos, audio outputs, signage, mechanical devices such as controllable 
mannequins, models, scent generators, or any other device configured to excite the 
human senses, in various combinations. The displays can be integrated with 
product arrangements, such as shelving, tables, counters and racks. 
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[019] Thus, by a 'display', we mean an output device that stimulates human 
senses. 

[020] The sensors can include, but are not limited to proximity sensors, 
microphones, thermal sensors, cameras, touch and motion sensors. The sensors 102 
are configured to determine consumer characteristics and behavior in the vicinity 
of the displays 101. Note that there is no need to explicitly identify the consumer. 
It is sufficient to detect broad classifications such as gender, height, weight, and 
age. Other sensors can also be integrated to provide environmental data such as 
weather, traffic, time, date, pricing, sales, etc. 

[021] The displays and sensors are connected to the processor by a network 
103. The network can be wired or wireless. 

System Operation 

[022] The basic operation of the processor 1 10 is shown in Figure 2. The 
system reads 201 sensor data 210, updates 202 a database 220, analyzes 203 the 
database, and updates 204 the displays 101 according to the analysis. The database 
can include preference models 221 for consumers, or classes of consumers as 
described in greater detail below. 

[023] It should be noted that the preference models according to the invention 
do no rely on explicit consumer identification, such as knowing the consumer 
name, account number, credit card number, etc. Instead, the preference models 
according to the invention are built from consumer characteristics which are 
relatively straightforward to detect, such as height, weight, gender, race, and the 
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like. This allows the system to respect the privacy of the individual. Thus, our 
system operates according to real consumer characteristics, e.g., gender and race, 
and not to some assigned identity, such as an account number. 

[024] Figure 3 shows a display with a projector as an element of the display 
system. Although video projectors are typically used for rendering images on a 
planar display surface, it is also possible to project images onto three-dimensional 
structural elements such as shelves, walls, signage, and products. Multiple 
projectors can be used to increase the size of the displayed images. The images can 
be warped and blended so as to appear seamless on curved surfaces. The 
appearance of high-resolution monitors can be simulated to provide detailed 
product information. Projected images can highlight product features, and visually 
show options such as color and texture. 

[025] Some of the sensors acquire real-time data reflecting consumer 
characteristics and behavior. A simple sensor is a proximity sensor for detecting a 
consumer's presence. These types of sensors can include passive IR sensors, active 
'break beam' sensors, microwave motion detectors and capacitive proximity 
sensors. If the sensors include cameras, the acquired images can be analyzed to 
locate the consumers, and even implicit characteristics of the consumer, such as 
height, weight, gender, and race characteristics, the number of consumers in a 
group, all using conventional vision pattern recognition and vision systems. 

[026] Herein, implicit characteristics means characteristics that are measured, 
such as weight or gender, as opposed to explicit identifications used in prior art 
systems, such as name or number. 
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[027] It should be noted that the system does not require explicit consumer 
identification, as in the prior art. For example, it can be determined which 
consumer is looking at which product, and the gender of the consumer. The images 
can also determine trajectories of consumers in the retail environment, i.e., what is 
the 'shopping pattern' of the consumer, is it quick, slow, directed, or random. The 
sensors can also detect heart rate and breathing rate. The sensed data is analyzed in 
real-time to determine an optimal presentation at any given moment, based on 
consumer characteristics such as height, gender, and age group rather than explicit 
knowledge of the individual. 

[028] For example, activated proximity sensors in a product or display shelf 
can initiate a particular stream of gender and age specific content at a display area 
on or near a product and the consumer. It is worth noting that the only action 
required of the consumer is normal shopping behavior. There are no devices for the 
consumer to carry and there is no explicit interface to manipulate or activate. 

[029] The sensed data, in combination with the preference model determine 
how the displays are updated. For example, if a consumer is observed leaving a 
product area, the display can be updated to entice the consumer back. 

[030] For example, the retail environment shown in Figure 3 includes a wall 
unit with shelves 301 for placing product. The projector 101 is suitably arranged so 
as to project on all of the shelves. Each shelf has an embedded conductive material 
302 that serves as a sense electrode for a capacitive proximity detector. These 
sense electrodes are connected to the processor 110, which allows the system to 
determine when a consumer is very near a shelf, as happens in the normal course 
of examining a product on a shelf. The display is updated to reflect the sensed data. 
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In addition, an audio display can play background music, narration, and other 
sound effects under control of the system. 

[031] The programmed interaction of this system can range from simple to very 
complex. In the simplest case, the content, e.g., audio and video content, is in the 
form of preprogrammed loops that play in response to activation by the shelf 
proximity sensors. A more complex version considers a history of interactions. For 
example, if the system detects a preponderance of recent interactions with one 
product in particular by a particular class of consumers, then the system can 
provide more detailed product information, or otherwise update the display in an 
attempt to "close the sale." If there have been no interactions for some period time, 
then the system can enter "attract mode," presenting content specifically designed 
to attract consumers to the display. 

[032] The system can select appropriate content so as to optimize the chance of 
a sale. This can be done by trying different content, and learning how the different 
content correlate to observed consumer responses. For example, the system can 
learn that certain "attract mode" displays work better at different times of day, or 
on different days of the week, depending on the class of consumers. The system 
can also correlate consumer responses to specific content. The ability to optimize 
the presentation in real-time through sensor observation and historical reference is 
one feature of the invention. 

[033] A display can include multiple computer-controlled output devices. In 
this case, the output images are blended to form a large, seamless display image. 
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Unlike the previous case where a certain area was set aside for the display, in this 
case, the projection fills the retail environment to a high degree. 

[034] The appearance of the space can thus be modified by simply changing the 
currently playing content. For example, at one moment, the store can appear to be a 
sophisticated, wood-paneled contemplative area, and at the next moment, it can 
appear to be a graffiti-covered brick alley. 

[035] By adding physically animated elements such as a moving wall, the 
effect of a morphable environment can be made even more realistic. In order to 
create undistorted displays, the geometry of the room is taken into account, and the 
projections suitably warped. 

[036] As in the case of the simple display, various theatrical, virtual signage, 
and virtual monitor effects can be created. However, in this example, the addition 
of a variety of sensors greatly enhances the interactive possibilities. 

[037] In particular, we add cameras throughout the environment to detect 
consumer presence, and to determine detailed consumer behavior and demographic 
and other characteristic information. This allows the system to determine when 
someone is looking at a display, determine their gender, race, approximate age, etc. 
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and then provide optimal content. The camera system can use any of a variety of 
signal processing algorithms such as face detection and gender classification. 

[038] Other types of sensors can prove useful in ascertaining consumer 
characteristics and behavior. Height and weight sensors can help the system to 
highlight appropriately sized items for a given consumer. Physiologic sensors can 
determine heart rate and other variables that correlate to consumer excitement. 

[039] Projectors can simulate theatrical lighting. For example, projecting bright 
circular areas has the effect of appearing as multiple spotlights. Similarly, various 
patterns can be projected to mimic gobos and other common lighting effects. Text 
and pictures can be projected on a wall or shelf surface to mimic signage. Video 
monitors are commonly used in retail environments to show moving video content. 
These can be simulated by a projector. In fact, a single projector can mimic 
multiple virtual monitors at various locations across a display. In addition, the 
virtual monitors can appear and disappear as needed, move, and have arbitrary 
shapes. The projection surface need not be flat. In many cases, it is desirable to use 
the product itself as the projection surface to point out features or give other 
information such as an internal view of the product. 

[040] By using multiple adjoining images, the appearance of a large portion of 
the environment can be changed, e.g., a wall can at one moment appear as wood- 
paneled, and at a next moment as a graffiti-covered brick alley. By adding 
physically animated elements such as a moving wall, the effect of a changing 
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environment can be made even more realistic. In order to generate undistorted 
displays, the geometry of the room is taken into account, and the projections can be 
suitably warped. 

[041] The database can also store real-time sales data, RFID information 
associated with products, inventory levels, and pricing and margin information. All 
of this data can be included in the analysis to determine the optimal way to update 
the display. 

[042] In addition to fixed content loops, the system can also include 
parameterizable content which can be programmatically modified in real-time as 
part of the overall optimization. Examples include the ability to occasionally have 
limited-time specials, where pricing is determined as part of the statistical 
optimization. 

[043] Although the invention has been described by way of examples of 
preferred embodiments, it is to be understood that various other adaptations and 
modifications can be made within the spirit and scope of the invention. Therefore, 
it is the object of the appended claims to cover all such variations and 
modifications as come within the true spirit and scope of the invention. 
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